fix(jira): add missing _raw_data_* columns to _tool_jira_sprint_reports by DoDiODev · Pull Request #9015 · apache/devlake

DoDiODev · 2026-07-28T08:28:08Z

Starting point is the Jira Sprint Report bug below. The regression guard added
for it turned out to catch the same class of bug in three more plugins
(taiga, teambition, testmo), which are fixed here as well — see
§3.

Problem

Collecting Jira data fails in the extractSprintReport subtask with:

subtask extractSprintReport ended unexpectedly
  error getting batch from result (500)
  Error 1054 (42S22): Unknown column '_raw_data_table' in 'where clause'

Root cause

The Sprint Report feature (PR #8967 / #9010) added the table
_tool_jira_sprint_reports. Its migration
(20260722_add_sprint_report_table.go) creates the table from a struct that
does not embed common.NoPKModel:

type jiraSprintReport20260722 struct {
    ConnectionId uint64 `gorm:"primaryKey"`
    // ... no common.NoPKModel ...
}

The runtime model models.JiraSprintReport does embed common.NoPKModel
(→ common.RawDataOrigin), so it expects the columns _raw_data_params,
_raw_data_table, _raw_data_id, _raw_data_remark (plus created_at,
updated_at).

During extraction, api.NewApiExtractor deletes outdated rows via
WHERE _raw_data_table = ? AND _raw_data_params = ?
(helpers/pluginhelper/api/batch_save_divider.go). Because the migration never
created those columns, MySQL rejects the query with error 1054.

What changed

1. Fix for the reported bug (jira)

New migration plugins/jira/models/migrationscripts/20260727_add_raw_data_columns_to_sprint_report.go
re-runs AutoMigrateTables on a struct that embeds the raw-data columns,
adding them to existing _tool_jira_sprint_reports tables without data
loss (GORM AutoMigrate only adds missing columns). Registered in
models/migrationscripts/register.go.
The original 20260722_add_sprint_report_table.go is left unchanged:
migration scripts are append-only, and editing it would not repair databases
that already ran it (its version is already recorded in
_devlake_migration_history).

2. Regression tests (schema-drift guards)

plugins/jira/e2e/migration_schema_test.go — runs the real Jira migration
chain (framework + jira) and asserts every column each Jira model declares
exists in the migrated table. Directly reproduces and guards the bug above.
plugins/schema_e2e/migration_schema_test.go — cross-plugin generalization:
applies framework + all plugin migrations and validates model-vs-table
column parity for every built-in Go plugin. Includes TestAllGoPluginsListed,
which fails if a new plugin directory with an impl package is added but not
registered, so the guard stays complete automatically.

Both deliberately run the real migration scripts instead of AutoMigrate-ing
the runtime model — an AutoMigrate-based check could never detect this class of
drift. Both live in e2e packages, so they run under make e2e-test-go-plugins
(require E2E_DB_URL) and are excluded from the DB-less unit-test run.

Implementation details worth noting:

Both tests call dalgorm.Init(...) to register the encdec GORM serializer.
runner.CreateBasicRes does not do this (only CreateAppBasicRes does),
so without it the migrations abort with invalid serializer type encdec.
Both fall back to a deterministic ENCRYPTION_SECRET if none is configured;
some migrations (e.g. jira 20220716) refuse to run without one
(jira v0.11 invalid encKey), and CI does not provide a value.
Models that no migration materializes (pure API-response models such as
_tool_jira_server_infos) are skipped — the check targets drift between an
existing table and its model.

3. Additional schema drifts found by the new cross-plugin guard

The guard immediately uncovered three pre-existing bugs of exactly the same
class. Each is fixed with a new, additive migration (registered in the
respective register.go):

Table	Missing column(s)
`_tool_taiga_scope_configs`	`type_mappings`
`_tool_teambition_scope_configs`	`id`, `created_at`, `updated_at`
`_tool_testmo_scope_configs`	`connection_id`, `name`

Files: plugins/{taiga,teambition,testmo}/models/migrationscripts/20260727_add_missing_scope_config_columns.go.

All new migration scripts use core/models/migrationscripts/archived — importing
core/models/common from a migration script is rejected by
make migration-script-lint.

Why these repairs are safe on populated tables

The scope-config repairs re-add columns that carry constraints in
common.ScopeConfig (name has a uniqueIndex) and, for teambition, an
AUTO_INCREMENT primary key. Both were verified against live engines rather than
assumed:

uniqueIndex on name — GORM adds new columns as nullable
(ALTER TABLE ... ADD COLUMN name VARCHAR(255), no NOT NULL DEFAULT ''), so
pre-existing rows are backfilled with NULL, and both MySQL and PostgreSQL
permit duplicate NULLs in a unique index. Creating the index on a table with
two pre-existing rows succeeded on MySQL 8.4.10 and PostgreSQL 17.2.
The counter-test confirms the distinction: with an explicit
NOT NULL DEFAULT '' column the same index fails
(Error 1062 / pq: … (23505)) — which is exactly the case that does not
occur here.
AUTO_INCREMENT id on the primary-key-less _tool_teambition_scope_configs —
GORM issues a plain ADD COLUMN, and MySQL backfills consecutive ids for the
existing rows (verified: two pre-existing rows received id 1 and 2). No
Error 1075 ("there can be only one auto column and it must be defined as a
key").

4. Build script

scripts/build-plugins.sh builds every directory under plugins/ with
-buildmode=plugin. plugins/schema_e2e/ is not a plugin (it contains only the
cross-plugin test and has no main package), which made make build-plugin fail
with "-buildmode=plugin requires exactly one main package". The directory is now
excluded, alongside the existing core / helper / logs exclusions.

Why a new migration (not editing the old one)

The buggy migration 20260722 is already on main/upstream/main and has
already been applied to real databases; its version is recorded, so it will
never re-run. Only a new migration can repair those databases.
Append-only migrations keep the migration history reproducible across
environments.

Testing

make migration-script-lint — OK.
gofmt -l on all added/changed files — clean.
go vet ./plugins/{jira,taiga,teambition,testmo,schema_e2e}/... — OK.
MySQL 8.4.10, fresh database:
- go test ./plugins/schema_e2e/ — OK, 42/42 plugins PASS
  (TestAllGoPluginsListed included).
- go test -run TestMigrationSchema ./plugins/jira/e2e/ — OK.
PostgreSQL 17.2, fresh database: both tests OK; the repaired scope-config
tables contain the added columns.
Upgrade path on a populated database (not just a fresh one): all four
migrations were applied to a real, long-running DevLake instance. They are
recorded in _devlake_migration_history and the target tables carry the
expected columns afterwards — e.g. _tool_teambition_scope_configs gained
id, created_at, updated_at, and _tool_jira_sprint_reports (661 rows)
gained the _raw_data_* columns without data loss.
Constraint safety on populated tables (unique index / AUTO_INCREMENT PK)
verified separately against MySQL 8.4.10 and PostgreSQL 17.2 — see
§3 "Why these repairs are safe on populated tables".
Negative test: dropping _raw_data_table from _tool_jira_sprint_reports
makes the cross-plugin guard fail with
[jira] table "_tool_jira_sprint_reports" is missing column "_raw_data_table"
— i.e. the guard genuinely detects the original regression.

Known limitations of the guard (intentional)

It checks column presence only — not column type, length, nullability,
primary keys or indexes.
It runs against the shared E2E database. Other plugin E2E tests use
FlushTabler (drop + AutoMigrate of the runtime model), so a table touched by
an earlier test can appear "repaired". On a fresh database (as in CI) this does
not apply.
Tables that no migration creates are skipped, so "model listed in
GetTablesInfo() but no table at all" is not flagged.

Integriert den Inhalt von PR apache#9015 in den lokalen Integrations-Branch. Die Sprint-Report-Migration 20260722_add_sprint_report_table.go legt _tool_jira_sprint_reports aus einem Struct an, das common.NoPKModel nicht einbettet, waehrend das Laufzeitmodell models.JiraSprintReport es einbettet. Die Spalten _raw_data_params / _raw_data_table / _raw_data_id / _raw_data_remark (plus created_at, updated_at) fehlten daher, sodass die Cleanup-Query des ApiExtractors WHERE _raw_data_table = ? AND _raw_data_params = ? den Subtask extractSprintReport mit "Error 1054 (42S22): Unknown column '_raw_data_table' in 'where clause'" abbrechen liess. Neue, additive Migration statt Aenderung der alten: Migrationsskripte sind append-only, und ein Edit wuerde Datenbanken nicht reparieren, die die Version bereits protokolliert haben. Zwei Schema-Drift-Guards, die die ECHTEN Migrationsskripte ausfuehren (statt das Laufzeitmodell zu AutoMigrate-n, was genau diese Drift verdecken wuerde): * plugins/jira/e2e/migration_schema_test.go - Jira-spezifisch. * plugins/schema_e2e/migration_schema_test.go - plugin-uebergreifend fuer alle Go-Plugins, inkl. TestAllGoPluginsListed. Der uebergreifende Guard deckte drei bestehende Drifts derselben Klasse auf, je mit eigener additiver Migration behoben: * _tool_taiga_scope_configs - type_mappings fehlte * _tool_teambition_scope_configs - id, created_at, updated_at fehlten * _tool_testmo_scope_configs - connection_id, name fehlten Verifiziert gegen MySQL 8.4.10 und PostgreSQL 17.2: neue Spalten werden nullable angelegt, doppelte NULLs sind im uniqueIndex zulaessig, und die AUTO_INCREMENT-PK laesst sich auf der PK-losen Teambition-Tabelle nachruesten (kein Error 1075). Alle vier Migrationen wurden zusaetzlich auf der bestehenden lokalen Datenbank angewendet. Ausserdem: plugins/schema_e2e in scripts/build-plugins.sh ausgeschlossen (kein Plugin, kein main-Package -> "make build-plugin" scheiterte mit "-buildmode=plugin requires exactly one main package"). AGENTS.md um die Schema-Drift-Guards und die zugehoerigen Fallstricke ergaenzt (fork-only, nicht Teil des Upstream-PRs). Signed-off-by: DoDiODev <DoDiDev@proton.me>

klesh · 2026-07-30T16:11:31Z

Hi. Could you please take a look at the failing test cases. Thanks.

The Sprint Report migration 20260722_add_sprint_report_table.go creates _tool_jira_sprint_reports from a struct that does not embed common.NoPKModel, while the runtime model models.JiraSprintReport does. The columns _raw_data_params / _raw_data_table / _raw_data_id / _raw_data_remark (plus created_at, updated_at) were therefore never created, so the ApiExtractor cleanup query WHERE _raw_data_table = ? AND _raw_data_params = ? made the extractSprintReport subtask fail with "Error 1054 (42S22): Unknown column '_raw_data_table' in 'where clause'". Add a new, additive migration that re-runs AutoMigrateTables on a struct embedding archived.NoPKModel. The original migration is left untouched: migration scripts are append-only, and editing it would not repair databases that already recorded its version. Add two schema-drift regression guards that run the REAL migration scripts instead of AutoMigrate-ing the runtime model, which would hide this class of drift: * plugins/jira/e2e/migration_schema_test.go - Jira-specific guard. * plugins/schema_e2e/migration_schema_test.go - cross-plugin guard for every built-in Go plugin, including TestAllGoPluginsListed so the guard stays complete when a new plugin is added. Both guards run the migrations against a dedicated, empty database created by the new helper e2ehelper.NewIsolatedMigrationDb: the shared e2e database is polluted by the other e2e tests, which AutoMigrate tables without recording anything in _devlake_migration_history, so running the real scripts against it fails with errors such as "Table 'cicd_pipeline_commits' already exists". The cross-plugin guard immediately uncovered three pre-existing drifts of the same class, each fixed with its own additive migration: * _tool_taiga_scope_configs - missing type_mappings * _tool_teambition_scope_configs - missing id, created_at, updated_at * _tool_testmo_scope_configs - missing connection_id, name The teambition table has no primary key at all, and its missing `id` is an auto-increment primary key, which AutoMigrate cannot append to an existing table (MySQL: "Incorrect table definition; there can be only one auto column and it must be defined as a key"). That column is therefore added with explicit DDL, which also keeps the ids of existing rows and the sequence/counter in sync on both MySQL and PostgreSQL. Finally, exclude plugins/schema_e2e from scripts/build-plugins.sh: it is not a plugin and has no main package, which broke `make build-plugin` with "-buildmode=plugin requires exactly one main package". Signed-off-by: DoDiODev <DoDiDev@proton.me>

Follow-up fixes from the review of apache#9015: - add e2ehelper.NewIsolatedMigrationDb so the schema-drift guards run the real migration scripts against a dedicated, empty database instead of the shared E2E_DB_URL one - use it in the jira and cross-plugin schema-drift tests - fix the teambition scope-config migration accordingly - document the findings in AGENTS.md

vbhanuchander-lang · 2026-08-01T22:50:20Z

Nice piece of work — particularly leaving 20260722_add_sprint_report_table.go untouched and repairing forward with an additive migration. That's the right call for anything already recorded in _devlake_migration_history, and it's the part people most often get wrong.

One gap worth considering in the new guard, since it's the most interesting part of this PR.

migration_schema_test.go runs the real migration chain against a fresh database and compares the resulting columns to the model. That catches drift in the end state, which is the bug you started from. But it runs against empty tables, so it cannot exercise the populated-table upgrade path — and that path is exactly where the teambition workaround lives:

id is an auto-increment primary key, which GORM's AutoMigrate cannot append to an existing table ... The column is therefore added with explicit DDL

On an empty _tool_teambition_scope_configs, plain AutoMigrate and the explicit DDL produce an identical schema, so the guard stays green either way. If someone later simplifies that DDL back to AutoMigrate because it looks redundant, the guard still passes and only real upgrades break — the same failure shape as the bug this PR fixes, one layer up.

Would it be worth inserting a row into each table before the tail migrations run, so the guard also asserts the migration applies to a non-empty table? Even a single row would have made the id case fail loudly rather than silently.

Not a blocker — the fix itself looks correct on both engines to me. Dialect() returns Dialector.Name(), so the "mysql" comparison matches, and BIGSERIAL PRIMARY KEY is the right Postgres counterpart.

DoDiODev · 2026-08-03T07:58:49Z

Thanks — I checked this against both engines rather than reasoning about it, and the specific example turns out to be already covered, for a slightly different reason than expected.

On MySQL 8.4.10, replacing the explicit DDL with plain AutoMigrate emits ALTER TABLE ... ADD id bigint unsigned AUTO_INCREMENT and fails with Error 1075 (42000): Incorrect table definition; there can be only one auto column and it must be defined as a key regardless of row count — it is a DDL-level rejection, so the guard fails on the empty table too (verified with 0 and with 2 pre-existing rows).

On PostgreSQL 17.2 the same AutoMigrate succeeds in both cases (ALTER TABLE ... ADD "id" bigserial), just without the PRIMARY KEY constraint — so a seeded row would not surface it there either. What is invisible on Postgres is the missing primary key, and a column-presence guard cannot see that with or without data.

Your general point stands, though: data-dependent failures are not exercised — NOT NULL without a default, a uniqueIndex over pre-existing duplicates, type narrowing. A fully generic "seed every table before the tail migrations run" would need a two-phase migrator run with an arbitrary version cutoff plus valid rows for every tool table (JSON columns, NOT NULL fields), which ages badly.

What I am adding instead is a targeted upgrade-path guard for the tables this PR repairs: build the pre-repair table shape, insert rows, run only the new script's Up(), then assert the added columns exist, the rows survived, and the ids were backfilled — plus a primary-key assertion, so the AutoMigrate-instead-of-DDL case is caught on Postgres as well. That codifies exactly what is currently only documented as manual verification.

Thanks for the careful read — the Dialect() / BIGSERIAL confirmation is appreciated.

…rt-raw-data-columns

The cross-plugin schema-drift guard added by this PR caught a fourth occurrence of the same bug class, introduced by apache#9019: _tool_copilot_enterprise_ai_credit_usage _tool_copilot_org_ai_credit_usage _tool_copilot_user_ai_credit_usage all lack gross_quantity, discount_quantity, net_quantity, price_per_unit, gross_amount, discount_amount and net_amount, while the runtime models models.GhCopilot{Enterprise,Org,User}AiCreditUsage declare them inline. Writing a record therefore fails with "Unknown column 'gross_quantity' in 'field list'". Root cause: 20260708_add_ai_credit_usage_metrics.go declares those seven columns through an anonymous embedded struct whose TYPE NAME IS UNEXPORTED creditUsageBreakdown20260708 `gorm:"embedded"` and GORM's schema parser skips anonymous fields of unexported types, so AutoMigrate never created the columns. Add a new, additive migration that AutoMigrates the missing columns. It only adds absent columns, so it is a no-op on databases that already have them and safe on populated tables. The original script is left untouched: migration scripts are append-only and its version is already recorded in _devlake_migration_history. Verified with the cross-plugin guard against a fresh database on MySQL 8.4.10 and PostgreSQL 17.2: 44/44 plugins pass (was 43/44 with gh-copilot failing on 21 missing columns). Signed-off-by: DoDiODev <DoDiDev@proton.me>

Review feedback on apache#9015: TestMigrationSchemaMatchesModels proves the END STATE of a fresh migration run matches the runtime models, but every table it inspects is empty, so it never exercises the upgrade path of a repair migration on a database that already holds rows -- which is the only situation those migrations exist for. Add TestMigrationUpgradePathOnPopulatedTables, which for every repair migration in this PR 1. recreates the table exactly as the buggy migration left it, 2. inserts rows, 3. runs ONLY that repair script, 4. asserts the columns were added, the rows survived, the table has a primary key and auto-increment ids were backfilled (plus that a subsequent INSERT still works, i.e. the sequence/counter is in sync). Step 4 covers what a column-presence check cannot see. Negative test, with the explicit AUTO_INCREMENT DDL in the teambition script replaced by a plain AutoMigrate: MySQL -> FAIL, migration errors out (Error 1075) PostgreSQL -> FAIL, "table has no primary key after ..." (AutoMigrate happily adds `bigserial` without a key, so this is invisible to the column-only guard) Covered: _tool_jira_sprint_reports, _tool_taiga_scope_configs, _tool_teambition_scope_configs, _tool_testmo_scope_configs and the three _tool_copilot_*_ai_credit_usage tables. The scripts are looked up through each plugin's own MigrationScripts() by version, so the test fails if one is removed or renumbered. Verified on MySQL 8.4.10 and PostgreSQL 17.2: 51/51 subtests pass (44 plugins + 7 upgrade-path cases). Signed-off-by: DoDiODev <DoDiDev@proton.me>

DoDiODev · 2026-08-03T11:52:23Z

Implemented as announced, plus the CI question from @klesh is now answered — both in one update.

1. Targeted upgrade-path guard on populated tables

New: backend/plugins/schema_e2e/migration_upgrade_path_test.go →
TestMigrationUpgradePathOnPopulatedTables.

For every repair migration in this PR it

recreates the table exactly as the buggy migration left it,
inserts rows,
runs only that repair script's Up() (looked up through the plugin's own MigrationScripts() by version, so the test breaks if a script is removed or renumbered),
asserts: the columns were added, the rows survived, the table has a primary key, auto-increment ids were backfilled with distinct non-zero values, and a subsequent INSERT still works (i.e. the sequence/counter is in sync).

Covered (7 cases): _tool_jira_sprint_reports, _tool_taiga_scope_configs, _tool_teambition_scope_configs, _tool_testmo_scope_configs and the three _tool_copilot_{enterprise,org,user}_ai_credit_usage tables.

Negative test — it really does catch the regression

Replacing the explicit AUTO_INCREMENT DDL in the teambition script with a plain AutoMigrate (exactly the "someone simplifies this later" scenario @vbhanuchander-lang described):

engine	result
MySQL 8.4.10	FAIL — `migration "add missing id/created_at/updated_at columns …" failed on a populated "_tool_teambition_scope_configs"` (Error 1075)
PostgreSQL 17.2	FAIL — `table "_tool_teambition_scope_configs" has no primary key after …`

The Postgres case is the one the column-presence guard cannot see: AutoMigrate happily adds bigserial without a key and reports success. That gap is now closed by the primary-key assertion.

2. Rebase + a fourth instance of the same bug class

Rebased onto current main (10 upstream commits). The guard immediately failed for gh-copilot: the three tables added by #9019 are missing all seven credit-breakdown columns.

table	missing
`_tool_copilot_enterprise_ai_credit_usage`	`gross_quantity`, `discount_quantity`, `net_quantity`, `price_per_unit`, `gross_amount`, `discount_amount`, `net_amount`
`_tool_copilot_org_ai_credit_usage`	same 7
`_tool_copilot_user_ai_credit_usage`	same 7

Root cause: 20260708_add_ai_credit_usage_metrics.go declares them via an anonymous embedded struct whose type name is unexported

creditUsageBreakdown20260708 `gorm:"embedded"`

GORM's schema parser skips anonymous fields of unexported types, so AutoMigrate never created the columns — while the runtime models declare them inline. The extractor would fail with Unknown column 'gross_quantity' in 'field list': the Jira Sprint Report failure mode, one plugin over.

Fixed the same way as the other three — a new, additive migration (20260731_fix_ai_credit_usage_breakdown_columns.go); the original script is untouched. The two new plugins (clickup, incidentio) were registered in allGoPlugins(), as TestAllGoPluginsListed demanded — the self-completing property working as intended.

3. CI

The workflows on this PR sit at action_required (fork PR awaiting maintainer approval), so I mirrored the same checks in my fork on an identical tree — all green:

lint (go), migration-script-lint, unit-test, config-ui, ASF header check, Grafana dashboard check
e2e (mysql) (make e2e-test-go-plugins + make e2e-test), including:
- --- PASS: TestMigrationSchema (Jira-specific guard)
- --- PASS: TestAllGoPluginsListed
- --- PASS: TestMigrationSchemaMatchesModels — 44/44 plugins
- --- PASS: TestMigrationUpgradePathOnPopulatedTables — 7/7 cases

Locally also verified against MySQL 8.4.10 and PostgreSQL 17.2: 51/51 subtests pass on both.

@klesh this should address the failing checks — happy to have the workflows approved on the PR itself to confirm on your infrastructure.

klesh

LGTM
Thanks for your contribution.

DoDiODev force-pushed the pr/jira-sprint-report-raw-data-columns branch from d2491ee to db538d3 Compare July 31, 2026 09:05

DoDiODev marked this pull request as draft July 31, 2026 18:20

DoDiODev added 2 commits August 3, 2026 12:09

Merge remote-tracking branch 'upstream/main' into pr/jira-sprint-repo…

d9e7376

…rt-raw-data-columns

DoDiODev force-pushed the pr/jira-sprint-report-raw-data-columns branch from 47cb3b5 to e400f66 Compare August 3, 2026 10:57

DoDiODev marked this pull request as ready for review August 3, 2026 11:52

klesh approved these changes Aug 3, 2026

View reviewed changes

klesh merged commit 352f3b5 into apache:main Aug 3, 2026
10 checks passed

DoDiODev deleted the pr/jira-sprint-report-raw-data-columns branch August 3, 2026 13:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(jira): add missing _raw_data_* columns to _tool_jira_sprint_reports - #9015

fix(jira): add missing _raw_data_* columns to _tool_jira_sprint_reports#9015
klesh merged 4 commits into
apache:mainfrom
DoDiODev:pr/jira-sprint-report-raw-data-columns

DoDiODev commented Jul 28, 2026

Uh oh!

klesh commented Jul 30, 2026

Uh oh!

vbhanuchander-lang commented Aug 1, 2026

Uh oh!

DoDiODev commented Aug 3, 2026

Uh oh!

DoDiODev commented Aug 3, 2026

Uh oh!

klesh left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

DoDiODev commented Jul 28, 2026

Problem

Root cause

What changed

1. Fix for the reported bug (jira)

2. Regression tests (schema-drift guards)

3. Additional schema drifts found by the new cross-plugin guard

Why these repairs are safe on populated tables

4. Build script

Why a new migration (not editing the old one)

Testing

Known limitations of the guard (intentional)

Uh oh!

klesh commented Jul 30, 2026

Uh oh!

vbhanuchander-lang commented Aug 1, 2026

Uh oh!

DoDiODev commented Aug 3, 2026

Uh oh!

DoDiODev commented Aug 3, 2026

1. Targeted upgrade-path guard on populated tables

Negative test — it really does catch the regression

2. Rebase + a fourth instance of the same bug class

3. CI

Uh oh!

klesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants